Emit host-side synchronization for GPU-to-host copies - #2490
Draft
ThrudPrimrose wants to merge 16 commits into
Draft
Emit host-side synchronization for GPU-to-host copies#2490ThrudPrimrose wants to merge 16 commits into
ThrudPrimrose wants to merge 16 commits into
Conversation
Ranks of one job derive the same build folder, so ranks that each compile build on top of each other and can load a library another rank is still writing. Eight processes running one GPU test out of one folder failed six times; with a folder each, none. The new cache_distaware config entry names the build cache root after the rank the launcher (MPI, Flux, Slurm) advertises. It is off by default, because sharing one build is also a valid setup: distributed_compile has rank 0 build and every other rank load its folder. That path now pins the broadcast folder on the ranks that hold the SDFG, the others being free to pass None.
Changed default value of cache_distaware from false to true.
Ranked-vs-shared assertions assumed distaware defaulted off, so clearing the env override fell through to the new true default and compared a rank-suffixed path against itself. Wrap the old off-path assertions in an explicit distaware=False context and add structural per-rank-root assertions for the new on-by-default behavior, for every cache mode.
The exact-leaf assertion assumed cache mode 'name'. A workflow whose DACE_cache resolves to anything else (env or a persisted config value the unlaunched fixture does not clear) flipped the leaf to a hash suffix and broke the path match. Pin it explicitly like the sibling tests pin their env, same env-wins-over-config precedence used to root-cause the distaware default flip.
All workers/ranks see every GPU and pile CUDA contexts onto device 0, which flakes as invalid device ordinal (101) under -n 32 on cscs CI.
The synchronization site after a device-to-host copy was a bare pass claiming the destination emitted it, and the destination-side helper early-returns on any node that stream assignment stamped, so nothing was emitted at all. Pageable host memory hides this because the driver blocks on it anyway; pinned memory, HIP and graph capture do not. Emit the StreamSynchronize where the copy is issued, force it for any host-located destination regardless of the stamp, and only fall back to it for a cross-stream consumer that stream assignment left without an event.
__dace_init_cuda returns early, without constructing the gpu_context, when no GPU-capable device is present. The init function only checked __result after it had already run the environment init code, the SDFG-level init code and every persistent allocation -- and a persistent GPU array allocates through DACE_GPU_CHECK, which dereferences the gpu_context that was never constructed. Check __result right after the target initializers. The later check stays, because the code it guards can fail on its own.
ThrudPrimrose
added a commit
that referenced
this pull request
Aug 11, 2026
The experimental CUDA target waits for a device-to-host copy at its first host consumer, but rendered the wait through the stream manager's context array template (__state->gpu_context->streams[i]) while the copy itself was issued on the wired gpu_streams array element (gpu_streams[gpu_streams_idx(i)]). Waiting on any expression other than the one the copy was enqueued on orders nothing if the two ever diverge, and the emitted wait could not be matched to the copy's stream at all. This mirrors the legacy-target fix from PR #2490, which emits the StreamSynchronize on the copy's own stream expression. The wait now renders the producer's __dace_current_stream connector memlet through the same index-function machinery that spelled the copy's stream argument, so the two are textually and semantically the same stream object. Producers without a wired stream connector keep the stream manager fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A device-to-host copy is issued as an asynchronous
MemcpyAsyncon a stream, but the synchronization site after it in the CUDA code generator was a barepassdeferring to the destination, and the destination-side helper returns early on any node that stream assignment stamped with a_cuda_stream-- so no host synchronization was emitted anywhere. The shape that bites is trisolv: a device scalar is copied into host memory and the next kernel launch takes it by value, which the host reads while packing launch arguments; pageable host memory hides this because the driver blocks on it anyway, while pinned memory, HIP and graph capture do not.This emits the
StreamSynchronizewhere the copy is issued, forces it for any host-located destination regardless of the stream stamp, and narrows the surrounding predicate so a same-stream consumer -- which the stream already orders -- no longer requests one.tests/codegen/gpu_d2h_host_sync_test.pyasserts on the emitted code that the synchronization sits between the device-to-host copy and the kernel launch that reads it and names the copy's own stream, and that a device-to-device copy still gets no host wait.